Distributed Stochastic Gradient Descent with Compressed and Skipped Communication

This paper introduces CompSkipDSGD, a new algorithm for distributed stochastic gradient descent that aims to improve communication efficiency by compressing and selectively skipping communication. In addition compression, CompSkipDSGD allows both workers the server skip in any iteration of training process reserve it future iterations without significantly decreasing testing accuracy. Our experimental results on large-scale ImageNet dataset demonstrate can save hundreds gigabytes while maintaining similar levels accuracy compared state-of-the-art algorithms. The are supported theoretical analysis demonstrates convergence under established assumptions. Overall, could be useful reducing costs deep learning enabling use datasets models complex environments.